Object distance measurement method and system based on camera

ABSTRACT

Provided is an object distance measurement method based on a camera. The object distance measurement method may include: receiving an image captured through a camera of a vehicle; extracting a bounding box area of an object included in the image; estimating location change information of the camera for a predetermined time period; and calculating a distance to the object on the basis of the location change information of the camera.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to and the benefit of Korean PatentApplication No. 10-2021-0015054, filed on Feb. 2, 2021, and KoreanPatent Application No. 10-2021-0015053, filed on Feb. 2, 2021, thedisclosure of which is incorporated herein by reference in theirentirety.

BACKGROUND 1. Technical Field

The present disclosure relates to an object distance measurement methodand system based on a camera.

2. Discussion of Related Art

Recently, research and development are being actively conducted on anautonomous driving-related technology as well as various types of driverassistance systems. In order to apply such technologies to a motorway orurban environment, it is necessary to recognize various objects such assigns or obstacles on the road.

FIGS. 1A and 1B are diagrams for describing a method for calculating anobject distance from a camera image according to the related art.

Since the method according to the related art provides a 2D image basedon a single camera, one variable value needs to be estimated in order toprovide location information of a 3D object. Since a vehicle runs on theground, it is assumed that the position of the vehicle in a z-axisdirection is 0. As the height specification of a suspension type orbridge-type sign among signs, previously known height information, e.g.,5 m, is used. Furthermore, when inner parameter information and outerparameter information of the camera are known in advance, the distancefrom a pixel position of an image may be calculated as illustrated inFIG. 1A or 1B.

FIGS. 2A to 2C are diagrams for describing the case in which priorheight information cannot be used.

For example, Korean Patent No. 10-1706455 (Road Sign Detection-BasedDriving Lane Estimation Method and Apparatus) or Korean PatentApplication Laid-Open No. 10-2015-0049529 (Apparatus and Method forEstimating Location of Vehicle) discloses a method for calculating adistance by using mounting height specification information of a sign.

As illustrated in FIGS. 2A to 2C, however, the height specification of asingle-column or multi-column traffic sign has a range value. When asingle-column traffic sign and a multi-column traffic sign are mixed andinstalled, the height of the traffic sign is not regulated. Thus, it isimpossible to estimate a distance on the basis of the prior heightinformation.

In addition, Korean Patent No. 10-1724868 (Apparatus and Method forRecognizing Traffic Mark Based on Image) discloses a method forestimating a distance by matching a captured image with a DB. However,the distance which can be provided by this method has low accuracy dueto an information error of a GPS or the like.

Traffic lights provide information on the movements and directions ofvehicles, in order to efficiently control a traffic flow, and a drivercontrols his/her vehicle by recognizing the state of a traffic light.

An autonomous driving controller for helping a driver's driving alsorequires an image-based traffic light state recognition technology, andvarious techniques related to the state recognition technology aresuggested.

FIG. 3 is a photograph showing the situation in which there are multipletraffic lights on the road where an intersection and a crosswalk areconsecutively present.

When only a single traffic light is seen from a captured forward image,a signal may be determined through recognition. However, when theintersection and the crosswalk are consecutively present as shown inFIG. 3, it is necessary to determine which signal is an intersectionsignal and which signal is a crosswalk signal. Furthermore, in the caseof a five-way intersection, a traffic light for controlling a trafficflow of another road may be seen from the image.

In general, a driver preferentially recognizes a signal at the shortestdistance ahead. Therefore, in order to control a vehicle by usingtraffic light information, distance information from a traffic light isneeded.

SUMMARY

Various embodiments are directed to an object distance measurementmethod and system based on a camera, which can measure the location anddistance of an object on the basis of two images captured through acamera while a vehicle moves.

Also, various embodiments are directed to an object distance measurementmethod and system based on a camera, which can acquire the distancebetween a front camera and a traffic light lens and the rotation angleof the front camera on the basis of an image captured through the frontcamera, select the nearest traffic light ahead on the basis of thedistance and the rotation angle, and perform vehicle control accordingto the state information of the nearest traffic light ahead.

However, the problems to be solved by the present disclosure are notlimited to the above-described problems, and other problems may bepresent.

In an embodiment, an object distance measurement method based on acamera may include: receiving an image captured through a camera of avehicle; extracting a bounding box area of an object included in theimage; estimating location change information of the camera for apredetermined time; and calculating a distance to the object on thebasis of the location change information of the camera.

In an embodiment, an object distance measurement system based on acamera may include: a camera installed at the front of a vehicle, andconfigured to capture an image; a memory configured to store a programfor calculating a distance to an object on the basis of the imagecaptured by the camera; and a processor configured to execute theprogram stored in the memory. By executing the program, the processormay extract a bounding box area of the object included in the image,estimate location change information of the camera for a predeterminedtime, and then calculate the distance to the object on the basis of thelocation change information of the camera.

In order to solve the above-described problems, a computer program inaccordance with another aspect of the present disclosure is coupled to acomputer as hardware, executes the object distance measurement methodbased on a camera, and is stored in a computer readable recordingmedium.

The other details of the present disclosure are included in the detaileddescriptions and the drawings.

In accordance with the embodiments of the present disclosure, the objectdistance measurement system and method may provide distance informationwhen a plurality of traffic lights appears, thereby deciding prioritieswhich are to be reflected into a driving policy.

Therefore, although a remote sign is recognized, the object distancemeasurement system and method may decide an operation of the vehicle inpreference to a near sign, such that the vehicle can travel according toa road situation, which makes it possible to improve driving stability.

Furthermore, even before a road situation was changed but not yetreflected into a DB or map, the object distance measurement system andmethod may check signs according to the distance priorities thereof, andreflect the check result into a vehicle driving policy.

Furthermore, the object distance measurement system and method maymeasure a distance and angle of a traffic light by using a single image,and thus accurately distinguish a signal for controlling the trafficvolume at the current location in a crowded intersection or five-wayintersection, thereby improving the city driving performance andstability of a driving assistance system.

Furthermore, since the object distance measurement system and method canmeasure a distance from a single image, the measurement of the objectdistance measurement system and method is not affected by noise ofanother system, and the object distance measurement system and methodmay precisely determine the current location of a vehicle on the basisof the distance to a traffic light, thereby improving a localizationfunction.

The effects of the present disclosure are not limited to theabove-mentioned effects, and the other effects which are not mentionedherein will be clearly understood from the following descriptions bythose skilled in the art.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A and 1B are diagrams for describing a method for calculating anobject distance from a camera image according to the related art.

FIGS. 2A to 2C are diagrams for describing the case in which priorheight information cannot be used.

FIG. 3 is a photograph showing the situation in which there are multipletraffic lights on the road where an intersection and a crosswalk areconsecutively present.

FIG. 4 is a block diagram illustrating an object distance measurementsystem in accordance with an embodiment of the present disclosure.

FIG. 5 is a flowchart illustrating an object distance measurement methodin accordance with an embodiment of the present disclosure.

FIGS. 6A and 6B are diagrams illustrating the example in which abounding box area and a reference point are set in first and secondimages.

FIG. 7 is a diagram illustrating the situation in which the first andsecond images are acquired as the location of a vehicle is changed.

FIG. 8 is a diagram for describing a process of estimating the distancefrom an object on the basis of the first and second images.

FIG. 9 is a diagram illustrating an optical flow on a ground area.

FIG. 10 is a flowchart for describing a rotation angle estimation methodin accordance with an embodiment of the present disclosure.

FIG. 11 is a flowchart for describing a process of calculating thedistance to a ground area pixel.

FIG. 12 is a flowchart illustrating a vehicle control method inaccordance with an embodiment of the present disclosure.

FIG. 13 is a flowchart for describing a process of extracting a lensarea of a traffic light.

FIG. 14 is a diagram illustrating the size of a lens within a trafficlight.

FIG. 15 is a diagram for describing a process of calculating thedistance between a camera and a lens area.

FIG. 16 is a diagram showing an example of the lens area which ischanged for each rotation angle.

FIG. 17 is a flowchart for describing a process of estimating a rotationangle for a lens area.

DETAILED DESCRIPTION

The advantages and characteristics of the present disclosure and amethod for achieving the advantages and characteristics will be clearlydescribed through the following embodiments with reference to theaccompanying drawings.

However, the present disclosure is not limited to the followingembodiments, but may be implemented in various shapes different fromeach other, and the following embodiments are only provided to easilydeliver the purposes, configurations and effects of the presentdisclosure to those skilled in the art to which the present disclosurepertains. Therefore, the scope of the present disclosure is defined byclaims.

Terms used in this specification are used for describing exemplaryembodiments while not limiting the present invention. The terms of asingular form may include plural forms unless referred to the contrary.The meaning of ‘comprise’ and ‘comprising’ used in the specificationspecifies a component, step, operation, and/or element but does notexclude the presence or addition of other components, steps, operations,and/or elements. Throughout the specification, like reference numeralsrepresent the same components, and the term “and/or” includes each ofmentioned components and one or more combinations thereof. Althoughterms “first” and “second” are used to describe various components, thecomponents are not limited by the terms. The terms are used only todistinguish one element from another element. Therefore, a firstcomponent described below may be a second component within the technicalidea of the present disclosure.

Unless defined differently, all terms (including technical andscientific terms) used in this specification may be used as meaningswhich are commonly understood by those skilled in the art to which thepresent disclosure pertains. Furthermore, terms which are defined ingenerally used dictionaries are not ideally or excessively construedunless clearly and specifically defined.

FIG. 4 is a block diagram illustrating an object distance measurementsystem 100 in accordance with an embodiment of the present disclosure.

The object distance measurement system 100 in accordance with theembodiment of the present disclosure includes a camera 110, a memory 120and a processor 130.

The camera 110 is installed at the front of a vehicle and captures animage.

The memory 120 stores a program for calculating the distance to anobject on the basis of the image captured by the camera 110, and theprocessor 130 executes the program stored in the memory 120.

By executing the program, the processor 130 extracts a bounding box areaof an object included in the image, estimates a location change of thecamera for a predetermined time period, and then calculates the distanceto the object on the basis of information regarding the location changeof the camera.

Furthermore, when the object is a lens area of a traffic light, theobject distance measurement system 100 in accordance with the embodimentof the present disclosure stores, in the memory 120, a program fordeciding the nearest traffic light ahead on the basis of an imagecaptured by the camera 110.

By executing the program stored in the memory, the processor 130extracts the lens area of the traffic light from the image, calculatesthe distance between the camera and the lens area, estimates a rotationangle based on the camera with respect to the lens area of the trafficlight, decides the nearest traffic light ahead on the basis of thedistance and the rotation angle, and then controls the vehicle on thebasis of state information (e.g., traffic light information) of thenearest traffic light ahead.

Hereafter, a method performed by the object distance measurement system100 in accordance with the embodiment of the present disclosure will bedescribed with reference to FIGS. 5 to 11.

In the present disclosure, for convenience of understanding, the case inwhich an object is a sign installed on the road will be taken as anexample for description. However, the present disclosure is not limitedthereto. That is, the object includes various obstacles installed on theroad, front-side and rear-side vehicles, pedestrians and the like.

On the road, a plurality of signs are installed to inform a driver ofinformation or regulations for helping the driver's driving. In the caseof a road where a different regulation is applied to each lane or aheading direction is different for each lane, a plurality of signs aremounted on a column. In this case, the driver gives a higher priority toa sign at a short distance rather than a sign at a long distance, andsequentially recognizes the signs and decides a driving policy,according to the priorities.

Until now, however, only the coordinate of a suspension-type orbridge-type sign having a fixed height could be estimated through theprocess of estimating the coordinate of a 3D sign by using 2D imageinformation. That is, since a general sign is also mounted on a trafficlight, a tunnel and the like according to a road situation, the generalsign does not have a fixed height. In this case, a distance cannot bemeasured through the existing method.

For another example, two kinds of speed limit signs may be recognizedfrom one image in a section where the speed limit is changed. In thecase of an existing sign whose height is not decided, the distanceinformation between the vehicle and the sign cannot be provided. Thus, acontroller cannot decide the priorities of different pieces ofregulation information.

FIG. 5 is a flowchart illustrating an object distance measurement methodin accordance with an embodiment of the present disclosure.

It may be understood that steps illustrated in FIG. 5 are performed bythe object distance measurement system 100 based on a camera, but thepresent disclosure is not necessarily limited thereto. In the presentdisclosure, a vehicle may include not only an autonomous vehicle, butalso a vehicle in which an autonomous controller can be installed andoperated.

First, the system receives an image captured through a camera of thevehicle in step S110. At this time, the camera of the vehicle may be afront camera for driving assistance of the vehicle, but the presentdisclosure is not limited thereto.

In an embodiment, the system in accordance with the embodiment of thepresent disclosure receives a first image captured at a first locationof the camera, and receives a second image captured at a second locationof the camera after a predetermined time period has elapsed. As such,the system calculates the distance to an object on the basis of thefirst and second images captured at the first and second locations.

Then, the system extracts a bounding box area of the object included inthe images in step S120. In an embodiment, the system in accordance withthe embodiment of the present disclosure may recognize the object byusing an object detection method. The object detection method mayinclude outputting the image coordinate of the corresponding object tothe bounding box area.

Based on the image coordinate, the system sets a reference point in thebounding box area.

FIGS. 6A and 6B are diagrams illustrating an example in which thebounding box area and the reference point are set in the first andsecond images.

In an embodiment, a reference point P may be set to the nearest locationwithin the bounding box area on the basis of the driving direction ofthe vehicle. In the case of a sign, a left bottom point P of a boundingbox area thereof may be used as the reference point for distanceestimation.

Then, the system estimates a location change of the camera for apredetermined time period in step S130.

FIG. 7 is a diagram illustrating the situation in which the first andsecond images are acquired as the location of the vehicle is changed.FIG. 8 is a diagram for describing a process of estimating the distanceto an object on the basis of the first and second images.

In the present disclosure, the two images, acquired when the location ofthe vehicle is slightly changed as illustrated in FIG. 7, are used toestimate the distance to the object. Since the location of the vehicleis slightly changed when the first and second images are acquired, thechange in ground height of the camera and the rotation of the camera inpitch and roll directions have a small influence.

When the first location of the camera is set to the origin and thedriving direction of the vehicle is set to the y-axis in order toestimate the location change of the camera, the location change may besimulated as a 2D problem as illustrated in FIG. 8.

Then, the system acquires a first pixel coordinate value for a pixelcorresponding to the location of the object on the image plane of thefirst image, and calculates a first straight line equation passingthrough the first pixel coordinate value on the basis of the origin. Thefirst pixel coordinate value and the first straight line equation may beexpressed as Equation 1 and Equation 2 below, respectively.

$\begin{matrix}{p_{1} = \left( {y_{1},f} \right)} & \left\lbrack {{Equation}\mspace{14mu} 1} \right\rbrack \\{y = {\frac{f}{y_{1}}x}} & \left\lbrack {{Equation}\mspace{14mu} 2} \right\rbrack\end{matrix}$

In Equations 1 and 2, y₁ represents the physical horizontal location ofa pixel corresponding to the location (reference point) of the objectwithin the image plane of the first image, and f represents the focusinformation of the camera.

Then, the system estimates the coordinate value (v sin θ, v cos θ) ofthe second location on the basis of a predetermined time period t takenfor the camera to move from the first location to the second location, avehicle velocity v, and vehicle steering angle information θ at thefirst location.

Based on the coordinate value, the system acquires a second pixelcoordinate value for the pixel corresponding to the location of theobject within the image plane of the second image.

When a physical horizontal location corresponding to the location of theobject within the image plane of the second image is y₂ in the case thatthe vehicle rotates in the yaw direction, the second pixel coordinatevalue may be expressed as Equation 3 below.

$\begin{matrix}{p_{2} = {\begin{bmatrix}{\cos\;\phi} & {{- \sin}\;\phi} \\{\sin\;\phi} & {\cos\;\phi}\end{bmatrix}\begin{bmatrix}{y_{2} - {{vt}\;\sin\;\theta}} \\{{{vt}\;\cos\;\theta} + f}\end{bmatrix}}} & \left\lbrack {{Equation}\mspace{14mu} 3} \right\rbrack\end{matrix}$

After calculating the second pixel coordinate value, the systemcalculates a second straight line equation which passes through thecoordinate value of the second location and the second pixel coordinatevalue and is expressed as Equation 4 below, on the basis of thecoordinate value of the second location and vehicle steering angleinformation ϕ at the second location.

$\begin{matrix}{y = \frac{{\sin\;{\phi\left( {y_{2} - {{vt}\;\sin\;\theta}} \right)}} + {\cos\;{\phi\left( {{{vt}\;\cos\;\theta} + f} \right)}} - {{vt}\;\cos\;\theta}}{{\cos\;{\phi\left( {y_{2} - {{vt}\;\sin\;\theta}} \right)}} - {\sin\;{\phi\left( {{{vt}\;\cos\;\theta} + f} \right)}} - {{vt}\;\sin\;\theta}}} & \left\lbrack {{Equation}\mspace{14mu} 4} \right\rbrack\end{matrix}$

Then, the system may calculate the location of the object through thecoordinate of an intersection between the first and second straight lineequations, and the intersection coordinate may be expressed as Equation5 below. After calculating the intersection coordinate of the object,the system may calculate the distance to the object from the secondlocation as the current location on the basis of the intersectioncoordinate of the object.

$\begin{matrix}{{x = \frac{{{vt}\;\cos\;\theta} - {\alpha\;{vt}\;\sin\;\theta}}{\frac{f}{y_{1}} - \alpha}},{y = {\frac{f}{y_{1}}\frac{{{vt}\;\cos\;\theta} - {\alpha\;{vt}\;\sin\;\theta}}{\frac{f}{y_{1}} - \alpha}}},} & \left\lbrack {{Equation}\mspace{14mu} 5} \right\rbrack\end{matrix}$

In Equation 5, α represents the term.

In the above-described embodiment of the present disclosure, thedistance to the object is estimated on the basis of a single imagethrough the front camera on the 2D plane. In an embodiment of thepresent disclosure, a rotation angle in the yaw direction may be furtherconsidered, and calibrated and applied to the calculated distance to theobject.

In other words, the angle is not significantly changed in an imagecaptured by the camera when the vehicle travels on an even road.However, when the vehicle travels on an uneven road, the angle issignificantly changed in an image captured by the camera, even thoughthe height of the camera is slightly changed. Therefore, when thedistance to the object is estimated on the basis of the image capturedon the uneven road, a wrong result is highly likely to be estimated.

Thus, in the embodiment of the present disclosure, the rotation angleinformation in the yaw direction may be estimated and reflected tocalculate a more accurate distance to the object.

FIG. 9 is a diagram illustrating an optical flow on a ground area. FIG.10 is a flowchart for describing a rotation angle estimation method inaccordance with an embodiment of the present disclosure. FIG. 11 is adiagram for describing a process of calculating the distance to a groundarea pixel.

When receiving an image through the camera in step S210, the systemcalculates an optical flow for consecutive frames of the image in stepS220, calculates the distance to a ground area pixel from the camera foreach frame in step S230, and calculates a translational velocity as avertical location change value on the basis of a vehicle velocity and asteering angle in step S240.

The optical flow shows the movement of brightness patterns asillustrated in FIG. 9, when the brightness value of each pixel is notchanged in consecutive images.

General optical flow vectors v_(x) and v_(y) may be expressed asEquation 6 below.

$\begin{matrix}{\begin{bmatrix}v_{x} \\v_{y}\end{bmatrix} = {- {\begin{bmatrix}\frac{f}{Z} & 0 & \frac{- x}{Z} & \frac{- {xy}}{f} & \frac{f^{2} + y^{2}}{f} & {- y} \\0 & \frac{f}{Z} & \frac{- y}{Z} & {- \left( {f^{2} + y^{2}} \right)} & \frac{xy}{f} & x\end{bmatrix}\begin{bmatrix}t_{x} \\t_{y} \\t_{z} \\w_{x} \\w_{y} \\w_{z}\end{bmatrix}}}} & \left\lbrack {{Equation}\mspace{14mu} 6} \right\rbrack\end{matrix}$

In Equation 6, f represents the focal distance of the camera, x and yrepresent the locations of the ground area pixel in the consecutiveframes of the image, (t_(x), t_(y), t_(z)) represents the translationalvelocity, and (w_(x), w_(y), w_(z)) represents a rotation angularvelocity, and Z represents the distance to the ground area pixel fromthe camera.

Since the height of the ground surface is 0 in Equation 6, the distanceto the ground area pixel from the camera in a single image may becalculated as illustrated in FIG. 11, and the translational velocity maybe calculated from the vehicle velocity and the steering angle. Finally,the system may calculate the rotation angular velocity throughsimultaneous equations for optical flow vectors of three or more imagepoints in step S250.

After calculating the rotation angular velocity, the system may estimatea rotation angle through an integration operation, by usingpredetermined time information at the first and second locations, instep S260.

Then, the system calculates distances Z to the ground area pixel fromthe first and second locations on the basis of the rotation angle ρ andthe height information h_(cam) of the camera, in step S270.

Then, the system calculates a difference between the distances to theground area pixel from the first and second locations in step S280, andthen calibrates the distance to the object, calculated on the basis ofthe location change of the camera, by using the calculated distancedifference in step S290.

As such, the object distance measurement system in accordance with theembodiment of the present disclosure may estimate a change in locationof the camera by estimating a vertical rotation angle using the opticalflow information, and calibrate the distance information from an objecton the basis of the location change, thereby calculating more accuratedistance information.

Steps S110 to S290 in the above descriptions may be further divided intoadditional steps or combined into less steps, depending on embodimentsof the present disclosure. Furthermore, some steps may be omitted ifnecessary, and the order of the steps may be changed. Furthermore, thecontent of FIG. 4 may be applied to the contents of FIGS. 5 to 11, eventhough the content of FIG. 4 is omitted.

Hereafter, an object distance measurement method based on a camera,which is performed by the object distance measurement system, inaccordance with another embodiment of the present disclosure will bedescribed with reference to FIGS. 12 to 17. In particular, the objectdistance measurement method in accordance with the present embodimentmay include sensing the nearest traffic light on the basis of thedistance to an object, which is calculated through the above-describedembodiment, and controlling the vehicle.

FIG. 12 is a flowchart illustrating a vehicle control method inaccordance with an embodiment of the present disclosure.

It may be understood that steps illustrated in FIG. 12 are performed bythe object distance measurement system 100 based on a camera, but thepresent disclosure is not necessarily limited thereto. In the presentdisclosure, a vehicle may include not only an autonomous vehicle, butalso a vehicle in which an autonomous controller can be installed andoperated.

First, the system receives an image captured through a camera of thevehicle in step S310. At this time, the camera of the vehicle may be afront camera for driving assistance of the vehicle, but the presentdisclosure is not necessarily limited thereto.

In an embodiment, the system may recognize the state information of atraffic light from the image captured through the camera of the vehicleat the same time as the captured image is received, and provide thestate information together with the captured image. The stateinformation of the traffic light indicates instruction information basedon the color of the traffic light.

Then, the system extracts a lens area of the traffic light from theimage captured by the camera in step S320.

FIG. 13 is a flowchart for describing the process of extracting the lensarea of the traffic light.

The traffic light includes a lens configured to output light and ahousing configured to cover the lens, and the object distancemeasurement method in accordance with the embodiment of the presentdisclosure requires a process of extracting the lens area.

For this process, the system extracts a bounding box area correspondingto a traffic light area from the image in step S321.

Then, the system selects a seed point within the bounding box area instep S322. In an embodiment, the brightest point around the center pointof the bounding box area, i.e., the brightest point within apredetermined range from the center point may be selected as the seedpoint. Since the lens is a part from which light is outputted, the lensarea is brighter than the dark housing. In some cases, the sky may bebrighter than the lens, due to sunlight. Thus, a point around the centerpoint is selected as the seed point.

Then, the system extracts the lens area of the traffic light on thebasis of the difference between the pixel value of the selected seedpoint and a peripheral pixel value, in step S323.

More specifically, the system calculates a pixel value (hereafter, aperipheral pixel value) at a position moved in a predetermined directionfrom the selected seed point, in step S3231. For example, the peripheralpixel value is calculated while the position is moved in 8 directionsfrom the seed point as the current location.

Then, the system calculates the difference between the pixel value ofthe seed point and the peripheral pixel value, and determines whetherthe calculated difference exceeds a predetermined threshold value, instep S3232.

When the determination result indicates that the difference between thepixel values is equal to or less than the threshold value, the systemmay select a position corresponding to the peripheral pixel value as thelens area in step S3233, and reset the peripheral pixel value in thecorresponding direction to the seed point in step S3234. While movingthe position in the predetermined direction on the basis of thecorresponding seed point, the system repeats the above-describedprocess. In some embodiments, a plurality of seed points may be set, anda plurality of positions corresponding to peripheral pixel values, whichare selected as lens areas while the positions are moved in a pluralityof directions from the initial seed point at the same time, may be usedas the respective seed points. Therefore, as the above-described processis repeated, the lens area may be more rapidly decided.

On the other hand, when the determination result indicates that thedifference between the pixel values exceeds the threshold value, thesystem determines that the position corresponding to the peripheralpixel value is an outer area of the lens, in step S3235.

Since the brightness of the lens area is significantly different fromthe brightness of the housing, the lens area may be accurately extractedthrough the above-described process.

Then, the system calculates the distance between the camera and the lensarea of the traffic light in step S330.

FIG. 14 is a diagram illustrating the size of a lens within a trafficlight. FIG. 15 is a diagram for describing a process of calculating thedistance between a camera and a lens area.

Since a general traffic light only needs to be installed at a height of450 cm or more, the general traffic light does not have a fixed height.However, since the size of a lens is decided as illustrated in FIG. 14,the distance between the camera and the lens area may be calculatedthrough the size of the lens.

FIG. 15 is a vertical cross-sectional view illustrating that the trafficlight is in front of the vehicle.

First, the system acquires the coordinate values of first and secondpixel locations corresponding to the start point and the end point ofthe major axis for a lens area within the image plane of an imagecaptured through a camera. When intrinsic parameters (f and pixel size)of the camera are calculated through a calibration process in FIG. 15,the system may acquire coordinate values y₁ and y₂ corresponding to thestart and end points at which the traffic light lens appears in adirection perpendicular to the image plane.

Then, the system acquires height information h from the camera to thelens of the traffic light. At this time, since the camera is mounted onthe vehicle, the mounting height Z_(cam) of the camera may be consideredtogether.

Then, the system calculates a distance d between the camera and the lensarea on the basis of the distance f between the camera and the imageplane based on the intrinsic parameters of the camera, the predeterminedlens size information (0.3 m), the height information h, and thecoordinate values y₁ and y₂ of the first and second pixel locations.

The process of calculating the distance d between the camera and thelens area may be expressed as the following equations.

When the coordinate values y₁ and y₂ corresponding to the start and endpoints at which the traffic light lens appears in the directionperpendicular to the image plane are acquired, the coordinate values y₁and y₂ need to satisfy Equation 7 below according to the proportionalrelation.

f:y ₁ =d:h

f:y ₂ =d:h _(+0.3 m)  [Equation 7]

At this time, y₁ and y₂ represent values obtained by converting thepixel locations into physical values, f represents an intrinsicparameter of the camera, obtained through the calibration process, and drepresents the distance to the traffic light lens.

The proportional relation equation may be summarized and expressed asEquation 8 below.

fh=dy ₁

fh+0.3f=dy ₂  [Equation 8]

When Equation 8 is summarized again, the distance from the camera to thetraffic light lens may be calculated as follows:

$d = {\frac{0.3f}{y_{2} - y_{1}}.}$

After calculating the distance from the camera of the vehicle to thetraffic light lens, the system may select the nearest traffic lightahead on the basis of the calculated distance in step S350. Then, thesystem may control the vehicle on the basis of the state information ofthe nearest traffic light ahead in step S360.

That is, when a plurality of traffic lights are included in the image,the system may select the nearest traffic light on the basis of thecalculated distance information, decide the nearest traffic light as atraffic light which affects the driving state at the current location,and control the vehicle according to the state of the correspondingtraffic light.

Furthermore, the system may decide the nearest traffic light ahead byusing angle information between the vehicle and the traffic light aswell as the distance information in step S340. That is, the system maynot only use the distance information of the plurality of traffic lightswhich are consecutively located in the forward direction, but alsoconsider a traffic light included in a side portion of the image in asituation such as an intersection or five-way intersection, in order todecide the nearest traffic light head. The traffic light included in theside portion of the image does not matter when the vehicle travelsstraight. However, when the vehicle travels along a curved road, thenearest traffic light ahead may be more effectively decided.

FIG. 16 is a diagram showing an example of the lens area which ischanged for each rotation angle. FIG. 17 is a flowchart for describing aprocess of estimating a rotation angle for a lens area.

Since a general traffic light is horizontally installed, a rotationangle in a pitch direction is small. However, depending on the directionof the road or vehicle, a yaw value is changed as illustrated in FIG.16. When the vehicle travels straight, the lens area may be clearlyextracted. However, as the rotation angle increases, the lens area maybe covered by the visor of the traffic light.

In order to solve such a problem, the system in accordance with theembodiment of the present disclosure calculates the length of the majoraxis for the lens area in step S341, and calculates the length of theminor axis for the lens area in step S342. For this operation, thesystem finds the coordinates of the left end, the right end, the upperend, and the lower end of the edge of the lens area, calculates thelength of the major axis through the vertical length of the edge, andcalculates the length of the minor axis through the horizontal length ofthe edge.

Then, the system calculates the rotation ratio from the calculatedlengths of the major axis and the minor axis in step S343. In anembodiment, the rotation ratio may be calculated through an operation ofdividing the length of the minor axis by the length of the major axis.

Then, the system estimates the rotation angle by approximating thecalculated rotation ratio through an LUT (Look Up Table) composed ofsimulation values, in step S344.

As such, the system may decide the nearest traffic light ahead among Ntraffic lights on the basis of the calculated rotation angle and thedistance information, and control the vehicle on the basis of the stateinformation of the nearest traffic light ahead.

Through this process, the system in accordance with the embodiment ofthe present disclosure may distinguish a signal for controlling thevolume of traffic at the current location even in a five-wayintersection or crowded intersection where a plurality of traffic lightsappear. Thus, the system may improve city driving assistance performanceand stability by using proper signal information.

In the above description, steps S310 to S360 may be divided intoadditional steps or combined into less steps, depending on embodimentsof the present disclosure. Furthermore, some steps may be omitted, ifnecessary, and the order of the steps may be changed. Furthermore, thecontents of FIGS. 4 to 11 may be applied to the contents of FIGS. 12 to17, even though the contents of FIGS. 4 to 11 are omitted.

The above-described object distance measurement method in accordancewith the embodiment of the present disclosure may be implemented as aprogram (or application) and stored in a medium, so as to be executedthrough a computer as hardware which is coupled thereto.

The above-described program may include codes written by a computerlanguage such as C, C++, JAVA, Ruby or machine language, which can beread by a processor (CPU) of the computer through a device interface ofthe computer, in order to execute the above-described method which isimplemented as a program read by the computer. Such codes may include afunctional code related to a function defining functions required forexecuting the above-described methods, and include an executionprocedure-related control code required for the processor of thecomputer to execute the functions according to a predeterminedprocedure. Furthermore, such codes may further include additionalinformation required for the processor of the computer to execute thefunctions or a memory reference-related code indicating the position(address) of an internal or external memory of the computer, where amedium needs to be referred to. Furthermore, when the processor of thecomputer needs to communicate with another remote computer or server inorder to execute the functions, the codes may further includecommunication-related codes indicating how to communicate with anotherremote computer or server by using a communication module of thecomputer and which information or media to transmit duringcommunication.

The stored medium does not indicate a medium such as a register, cacheor memory, which stores data for a short moment, but indicates a mediumwhich semi-permanently stores data and can be read by a device.Specifically, examples of the storage medium include a ROM, RAM, CD-ROM,magnetic tape, floppy disk, optical data storage device and the like,but the present disclosure is not limited thereto. That is, the programmay be stored in various recording media on various servers which thecomputer can access or various recording media of a user's computer.Furthermore, the media may store codes which can be distributed incomputer systems connected through a network, and read by computers in adistributed manner.

The descriptions of the present disclosure are only examples, and bythose skilled in the art to which the present disclosure pertains willunderstand that the present disclosure can be easily modified into otherspecific forms without changing the technical spirit or essentialfeatures of the present disclosure. Therefore, it should be understoodthat the above-described embodiments are only illustrative in allaspects and are not limitative. For example, components described in asingular form may be distributed and embodied. Similarly, distributedcomponents may be embodied in a coupled form.

While various embodiments have been described above, it will beunderstood to those skilled in the art that the embodiments describedare by way of example only. Accordingly, the disclosure described hereinshould not be limited based on the described embodiments.

What is claimed is:
 1. An object distance measurement method,comprising: receiving an image captured by a camera of a vehicle;extracting, from the captured image, a bounding box area of an objectincluded in the captured image; estimating a location change of thecamera for a predetermined time period; and calculating a distance tothe object based on the estimated location change of the camera.
 2. Theobject distance measurement method of claim 1, wherein: receiving thecaptured image comprises: receiving a first image captured by the cameraat a first location; and receiving a second image captured by the cameraat a second location after the predetermined time period has elapsed,and calculating the distance to the object comprises calculating thedistance to the object based on the estimated location change of thecamera from the first location to the second location.
 3. The objectdistance measurement method of claim 2, wherein estimating the locationchange of the camera comprises: setting the first location of the cameraas an original location, and setting the driving direction of thevehicle as a point on a y-axis; and estimating a coordinate value of thesecond location based on (1) the predetermined time period, and (2)vehicle velocity and vehicle steering angle of the vehicle at the firstlocation.
 4. The object distance measurement method of claim 3, whereincalculating the distance to the object comprises: acquiring a firstpixel coordinate value for a first pixel corresponding to a firstlocation of the object on an image plane of the first image; calculatinga first straight line equation passing through the first pixelcoordinate value based on the original location; acquiring a secondpixel coordinate value for a second pixel corresponding to a secondlocation of the object on an image plane of the second image;calculating a second straight line equation passing through a coordinatevalue of the second location and the second pixel coordinate value basedon (1) the coordinate value of the second location, (2) the second pixelcoordinate value, and (3) vehicle steering angle of the vehicle at thesecond location; and calculating the distance to the object based on acoordinate of an intersection between the first and second straight lineequations.
 5. The object distance measurement method of claim 4, whereinthe first and second pixel coordinate values are respectively calculatedbased on focus information of the camera and a physical horizontallocation of a pixel corresponding to the first and second locations. 6.The object distance measurement method of claim 2, wherein calculatingthe distance to the object comprises: estimating a rotation angle of thecamera based on an optical flow of the image; calculating, based on therotation angle and height of the camera, first and second distancesrespectively from the first and second locations to a ground area pixel;calculating a difference between the first and second distances; andcalibrating the distance to the object based on the location change ofthe camera and the distance difference.
 7. The object distancemeasurement method of claim 6, wherein estimating the rotation angle ofthe camera comprises: estimating a rotation angular velocity based on afocal distance of the camera, a location of the ground area pixel on aplurality of consecutive frames of the image, the distance from thecamera to the ground area pixel, a translational velocity of thevehicle, and an optical flow vector of the camera; and estimating therotation angle for the estimated rotation angular velocity based on thepredetermine time period.
 8. The object distance measurement method ofclaim 1, further comprising: determining, based on the captured image,that the object is a lens area of a traffic light; selecting a nearesttraffic light ahead of the vehicle based on the calculated distance; andcontrolling the vehicle based on traffic light information of thenearest traffic light ahead.
 9. The object distance measurement methodof claim 8, wherein extracting the lens area of the traffic light fromthe image comprises: extracting, from the captured image, a bounding boxarea corresponding to the traffic light area; selecting a seed pointwithin the bounding box area; and extracting, from the bounding boxarea, a lens area of the traffic light based on a difference between apixel value of the selected seed point and a peripheral pixel value at apoint located remotely from the selected seed point in a predetermineddirection.
 10. The object distance measurement method of claim 9,wherein selecting the seed point within the bounding box area comprisesselecting, as the seed point, a brightest point within a predeterminedrange from a center of the bounding box area.
 11. The object distancemeasurement method of claim 9, wherein extracting the lens area of thetraffic light comprises: calculating the peripheral pixel value;calculating a difference between the pixel value of the seed point andthe peripheral pixel value; determining whether the calculateddifference exceeds a predetermined threshold value; in response todetermining that the calculated difference does not exceed the thresholdvalue, determining a position corresponding to the peripheral pixelvalue as the lens area; and resetting the peripheral pixel value to theseed point.
 12. The object distance measurement method of claim 11,wherein extracting the lens area of the traffic light comprises, inresponse to determining that the calculated difference exceeds thethreshold value, determining that the position corresponding to theperipheral pixel value is an outer area of the lens.
 13. The objectdistance measurement method of claim 8, wherein calculating the distanceto the object comprises: acquiring a first coordinate value of a firstpixel location corresponding to a start point of a major axis for thelens area on the captured image; acquiring a second coordinate value ofa second pixel position corresponding to an end point of a major axisfor the lens area within an image plane of the captured image; acquiringheight information indicating a height difference between the camera andthe lens area of the traffic light; and calculating a distance betweenthe camera and the lens area based on the distance between the cameraand the image plane, an intrinsic parameter of the camera, predeterminedsize information of the lens, height information, and the coordinatevalues of the first and second pixel locations.
 14. The object distancemeasurement method of claim 8, wherein estimating the location change ofthe camera comprises estimating a rotation angle of the camera withrespect to the lens area of the traffic light.
 15. The object distancemeasurement method of claim 14, wherein estimating the location changeof the camera comprises: calculating lengths respectively of a majoraxis and a minor axis of the lens area; calculating a rotation ratiofrom the calculated lengths of the major axis and the minor axis; andestimating the rotation angle by approximating the calculated rotationratio through a lookup table.
 16. A system for calculating a distance toan object based on an image captured by a camera positioned at a frontportion of a vehicle, the system comprising: a processor; and a memoryin communication with the processor and storing instructions that, whenexecuted by the processor, cause the processor to control the system toperform: extracting a bounding box area of the object included in thecaptured image; estimating a location change of the camera for apredetermined time period; and calculating a distance to the objectbased on the estimated location change of the camera.
 17. The system ofclaim 16, wherein the instructions, when executed by the processor,further cause the processor to control the system to perform: receivinga first image captured by the camera at a first location; receiving asecond image captured by the camera at a second location after thepredetermined time period has elapsed; extracting, from the first andsecond images, a bounding box area of an object included in the firstand second images; setting a reference point in the bounding box area;estimating a rotation angle of the camera based on an optical flow ofthe first and second images; and calculating a distance to the objectbased on the estimated location change and rotation angle of the camera.18. The system of claim 16, wherein the instructions, when executed bythe processor, further cause the processor to control the system toperform: calculating first and second distances from the first andsecond locations to each of a plurality of ground area pixels based theestimated rotation angle of the camera and height information of thecamera; calculating a difference between the first and second distances;and calibrating the distance to the object based on the estimatedlocation change of the camera and the calculated difference between thefirst and second distances.
 19. The system of claim 16, wherein theinstructions, when executed by the processor, further cause theprocessor to control the system to perform: determining that the objectis a lens area of a traffic light; and selecting a nearest traffic lightahead of the vehicle based on the calculated distance to the object; andcontrolling the vehicle based on traffic light information of thenearest traffic light ahead.
 20. A system for deciding a nearest trafficlight ahead of a vehicle based on an image captured by the camerapositioned at a front portion of the vehicle, comprising: a processor;and a computer-readable medium in communication with the processor andstoring instructions that, when executed by the processor, cause theprocessor to control the system to perform: extracting, from thecaptured image, a lens area of a traffic light; calculating a distancebetween the camera and the lens area; estimating a rotation angle of thecamera with respect to the lens area of the traffic light; deciding anearest traffic light ahead of the vehicle based on the calculateddistance and the estimated rotation angle; and controlling the vehiclebased on traffic light information of the nearest traffic light.